Surveying Data for Patchy Structure

The term “data surveying” refers to the preliminary examination of a dataset to assess its overall character, and this process typically involves simple descriptive statistics to characterize the available variables, along with detection of data anomalies (e.g., outliers or incomplete records) and possibly other “interesting” or “unusual” features that may be worthy of careful scrutiny. In the survey sampling literature, an important distinction is made between responses that are missing at random, the simplest form of ignorable missing data, and non-random alternatives that can lead to non-ignorable missing data. The distinction is practically important because non-ignorable missing data can cause severe biases in analytical results, while ignorable missing data typically causes an undesirable but less serious increase in the variability of these results. Analogous distinctions can also be usefully made for other types of data anomalies (e.g., i.i.d. vs. correlated outliers) or other unusual data subsets of potential interest. In particular, the observation of systematic behavior with respect to time, position, or other ordered index sequences (e.g., primary key in a database) can often give insight into the nature or generation mechanism of these data subsets. Motivated by these observations, this paper considers the problem of detecting structure in distinguished subsets of data records, including missing data, outliers and other “interesting data records.” Depending on the nature of the dependences considered, this problem is closely related to a number of others, including the detection of “streaks” in athletic performance records, the quantification of association between variables, or binary classification.